System and method for receiver-driven streaming in a peer-to-peer network

ABSTRACT

A “PeerStreamer” provides receiver-driven peer-to-peer (P2P) media streaming for loosely coupled P2P networks. Peers in the network perform only simple operations, may cache all or part of the streaming media, do not collaborate with other peers, may be unreliable, and may drop offline or come online during any given streaming session. Clients in the network operate in real-time to coordinate peers, stream media from multiple peers, perform load balancing, handle online/offline states of peers, and perform decoding and rendering the streaming media. In one embodiment, the PeerStreamer uses high rate erasure resilient coding to allow multiple serving peers to hold partial media without conflict, such that clients simply retrieve fixed numbers of erasure coded blocks regardless of where and what specific blocks are retrieved. In another embodiment, the PeerStreamer uses embedded coded media to vary streaming bitrates according to available serving bandwidths and client queue status.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of U.S. patent applicationSer. No. 10/934,823, filed on Sep. 3, 2004, by Jin Li and entitled “ASYSTEM AND METHOD FOR DISTRIBUTED STREAMING OF SCALABLE MEDIA.”

BACKGROUND

1. Technical Field

The invention is related to receiver-driven peer-to-peer (P2P) mediastreaming for loosely coupled P2P networks, and in particular, to asystem and method for streaming media from a plurality of peers to aclient under the real-time coordination and control of the clientwithout the need to provide peer-to-peer collaboration.

2. Related Art

Recent market research has indicated that over half of the Internetusers in the United States have accessed some form of streaming media in2004. Access to streaming music is a very popular activity, while thepopularity of streaming video is growing rapidly.

Unfortunately, unlike typical web pages, a streaming media file istypically extremely large in size. For example, a 3 minute movie trailerencoded at 2 megabits per second (Mbps) can result in a 45 megabyte (MB)media file depending upon the codec used. Another problem that must beaddressed by streaming media is the critical timing of packet delivery.Consequently, the large size of the streaming media files and the packetdelivery timing requirements cause typical streaming media servers to berelatively expensive to set up and run. For example, one currentestimate puts the going rate for streaming media at $10 per 1 GB ofserving traffic. Using the example of a 45 MB file size, this can resultin a bandwidth cost of $0.45 per movie trailer distributed. Clearly suchcosts can escalate rapidly as the amount of media streaming increases.

One solution to the relatively high cost of media streaming is to use a“peer-to-peer” (P2P) network to provide the media streaming toindividual clients. In general, the basic idea of P2P networks is toallow each peer node to assist the media server in distributing thestreaming media. The success of P2P networks for streaming media hasresulted in a large number of conventional approaches to implementingP2P networks.

For example, conventional P2P schemes referred to as “end systemmulticast” and “PeerCast” use application-level multicast (ALM) formedia streaming. In particular, with both ESM and PeerCast, the peernodes are self organized into an overlay tree over an existing IPnetwork. The streaming data is then distributed along the overlay tree.The cost of providing bandwidth is then shared amongst the peer nodes,thereby reducing the bandwidth burden (and thus dollar cost) of runningthe media server. However, with both ESM and PeerCast, the leaf nodes ofthe distribution tree only receive the streaming media and do notcontribute to content distribution.

Two other conventional schemes, “CoopNet” and “SplitStream” address thecontent distribution limitation of schemes such as ESM and PeerCast byusing multiple distribution trees that span the source and the peernodes. Each tree in CoopNet and SplitStream can then transmit a separatepiece of streaming media. As a result, all peer nodes can be involved incontent distribution.

Additional examples of conventional P2P media streaming solutionsinclude a streaming scheme referred to as “OStream.” OStream uses a“cache-and-relay” approach such that peer nodes can serve clients withpreviously distributed media from its cache. Another conventionalsystem, “GnuStream” provides a receiver driven P2P media streamingsystem built on top of the well known “Gnutella” system. Yet anotherconventional scheme, referred to as “CollectCast” actively looks forserving peers that are most likely to achieve a best streaming quality,while dynamically adapting to network fluctuations and peer failures.

Another type of conventional scheme provides a type of distributed filesharing where pieces of a file are widely distributed across a number ofpeers. Then whenever a client requests a download of that file, thatrequest is serviced from a plurality of peers rather then directly fromthe server. For example, one such scheme, referred to as “Swarmcast,”spreads the load placed on a web site offering popular downloadablecontent by breaking files into much smaller pieces. Once a user hasinstalled the Swarmcast client program, their computers automaticallycooperate with other users' computers by passing around (i.e., serving)pieces of data that they have already downloaded, thereby reducing theoverall serving load on the central server. A similar scheme, referredto as “BitTorrent,” works along very similar principles. In particular,when under low load, a web site which serves large files using theBitTorrent scheme will behave much like a typical http server since itperforms most of the serving itself. However, when the server loadreaches some relatively high level, BitTorrent will shift to a statewhere most of the upload burden is borne by the downloading clientsthemselves for servicing other downloading clients.

Unfortunately, while schemes such as Swarmcast and BitTorrent are veryuseful for distributing pieces of files for dramatically increasingserver capacity as a function of the P2P network size, these systems arenot adapted for efficiently streaming media. In particular, schemes suchas Swarmcast and BitTorrent do not care about the order or timing of thedelivery of data packets constituting the file or files beingdownloaded. The files are simply broadcast in pieces from various peersto a client, and then simply locally reassembled in the correct order toreconstruct the original file on the client computer. However, in thecase of streaming media, the timing and order of data packets must becarefully considered and controlled so as to provide for efficientstreaming of that media.

Therefore, what is needed is a system and method for receiver-drivencontrol of media streaming from a collection of loosely coupled peers toa client. Such a system should not require communication orcollaboration between peers. Further, such as system and method shouldminimize computation demands placed onto peers by requiring the clientto perform the bulk of any necessary computational operations.

SUMMARY

A “PeerStreamer,” as described herein provides receiver-drivenpeer-to-peer (P2P) media streaming for loosely coupled P2P networks.Peers in the network perform only simple operations, may cache all orpart of the streaming media, do not collaborate with other peers, may beunreliable, and may drop offline or come online during any givenstreaming session. Clients (or receivers) in the network operate inreal-time to coordinate peers, stream media from multiple peers, performload balancing, handle online/offline states of peers, and performdecoding and rendering the streaming media.

Note that while the PeerStreamer system described herein is applicablefor use in large P2P networks with multiple clients and peers, thefollowing description will generally refer to individual clients forpurposes of clarity of explanation. Those skilled in the art willunderstand that the described system and method offered by thePeerStreamer is applicable to multiple clients. In addition, as thepeers described herein are used to serve the media to the receiver orclient, the cluster of peers in the P2P network are generally referredto herein either as peers, or as “serving peers.” It should also benoted that these “serving peers” should not be confused with “mediaservers,” as described herein, from which particular streaming mediafiles initially originate.

In general, the PeerStreamer provides receiver-driven media streaming.PeerStreamer operations begin with each receiving client retrieving alist of nearby peers that hold all or part of the requested streamingmedia. Note that in this context, a media server can also act as one ofthe serving peers. This list includes the IP addresses and the listeningports of a set of one or more neighbor serving peers that hold acomplete or partial copy of the serving media. Methods for retrievingthis list include: 1) retrieving the list directly from the mediaserver; 2) retrieving the list from a known serving peer; and 3) using adistributed hash table (DHT) approach for identifying the serving peers.

Once the client has retrieved the list of available serving peers, theclient connects to each serving peer and obtains its “availabilityvector.” In general, the availability vector for each serving peer is acompact description of the exact portion of the media held by thatserving peer. These availability vectors are then used by the client todetermine exactly what blocks of the encoded media are held by thevarious serving peers.

For example, where a particular serving peer holds the entire servingmedia the availability vector of that peer can be a single flag thatindicates that the serving peer holds a complete media copy. Similarly,if the serving peer holds only a portion of the serving media, theavailability vector of that serving peer will signal to the client whatportion of the media is held by the serving peer, e.g., the number ofblocks of each packet and the block indexes that are held by the servingpeer.

Further, where additional coding is used, such as the erasure codingtechniques described below, the availability vector will include themedia erasure coding keys assigned to serving peer, and the number oferasure blocks held by the serving peer. In addition, if the servingpeer uses erasure coding and the media is also embedded coded, theavailability vector will include the assigned media erasure coding keys,the number of erasure blocks of each packet at the different bitratelevels used by the embedded coding.

In general, an encoded media file typically includes a “media header”followed by a number of media packets (i.e., the “media body”)representing the encoded media. Given the availability vector, the nextstep is for the client to retrieve the lengths of the media header and a“media structure” which are derived from the encoded media file to bestreamed from the peer cluster. The media structure of a set of packetsis simply the packet headers plus the packet bitstream lengths. Afterthese lengths have been retrieved, the client calculates “data unit IDs”of the media header and media structure, and retrieves them from one ormore peers in the peer cluster in a collaborative fashion.

Once the media header arrives, the client analyzes the media header, andthen configures or initializes whatever audio/video decoders andrendering devices that are needed for decoding and rendering or playingback the specific type of media being streamed (i.e., MPEG 1/2/4, WMA,WMV, etc.) Once this initial setup phase has been completed, the clientthen proceeds to coordinate ongoing streaming of the media body from thepeer cluster as described below.

In particular, given the aforementioned media structure of theparticular streaming media, the client calculates data unit IDs ofpackets of the streaming media (i.e., the media body), and thenretrieves those packets one by one. In a related embodiment, thePeerStreamer uses embedded coded media, and the streaming bitrates thenvary according to available serving bandwidths and client queue status.In this case, ongoing retrieval of media packets of the media bodycorresponds to those packets that will provide the minimum ratedistortion based on the available bandwidth.

In either case, the client periodically updates the serving peer list,and connects to potential new serving peers. In a tested embodiment, theclient checked for potential new serving peers by issuing periodicconventional TCP connect function calls for each potential serving peer.After the client establishes the connection to a new serving peer, itfirst retrieves the aforementioned availability vector. The new peer maythen join the other active peers in the cluster, at the direction of thereceiver/client. The client then coordinates the peers, balances theserving load of the peers according to their serving bandwidths andcontent availability, and redirects unfulfilled requests of disconnectedor timed-out peers to one or more of the other active peers. Thestreaming operation then continues in this manner until the entirestreaming media is received, or the streaming operation is stopped bythe user.

In one embodiment, the PeerStreamer uses high rate erasure resilientcoding to allow multiple serving peers to hold partial media withoutconflict, such that clients simply retrieve fixed numbers of erasurecoded blocks regardless of where and what specific blocks are retrieved.In this case, the received erasure coded blocks are deposited into astaging queue of the client, where the media packet is then assembled.Completely assembled media packets are then sent downstream to bedecoded and played back using whatever audio/video decoders andrendering devices have configured or initialized for decoding andrendering or playing back the specific type of media being streamed. Inthis case, by controlling the length of the staging queue, the length ofa request queue, and the length of a compressed audio/video buffer, theclient maintains a streaming buffer of some desired period (on the orderof about four seconds in a tested embodiment). This combined buffer isthen used to combat network packet loss and jitter.

In view of the above summary, it is clear that the PeerStreamerdescribed herein provides a unique system and method for providingreceiver-driven media streaming in a P2P network. In addition to thejust described benefits, other advantages of the PeerStreamer willbecome apparent from the detailed description which follows hereinafterwhen taken in conjunction with the accompanying drawing figures.

DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the present inventionwill become better understood with regard to the following description,appended claims, and accompanying drawings where:

FIG. 1 is a general system diagram depicting a general-purpose computingdevice constituting an exemplary system implementing a “PeerStreamer”,as described herein.

FIG. 2 illustrates an exemplary peer-to-peer (P2P) network forreceiver-driven media streaming, as described herein.

FIG. 3 provides an exemplary architectural flow diagram whichillustrates program modules for implementing the PeerStreamer, asdescribed herein.

FIG. 4 illustrates a file format of a streaming media file, as describedherein.

FIG. 5 illustrates “data units” used in a tested embodiment of the bythe PeerStreamer, as described herein.

FIG. 6 illustrates partial caching of an embedded coded media packetthat has been split into 8 data units, as described herein.

FIG. 7 provides a sample DirectShow™ filter graph of a clientsPeerStreamer media streaming session.

FIG. 8 provides an architectural system diagram representingPeerStreamer request and staging queues and streaming media decoding,rendering and playback, as described herein, with system buffers beingillustrated by dashed lines.

FIG. 9 provides a block diagram illustration of PeerStreamer clientstaging queues for arriving data units, and PeerStreamer client requestqueues for each serving peer.

FIG. 10 provides an operational flow diagram which illustrates thegeneral operation of one embodiment of the PeerStreamer, as describedherein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description of the preferred embodiments of the presentinvention, reference is made to the accompanying drawings, which form apart hereof, and in which is shown by way of illustration specificembodiments in which the invention may be practiced. It is understoodthat other embodiments may be utilized and structural changes may bemade without departing from the scope of the present invention.

1.0 Exemplary Operating Environment:

FIG. 1 illustrates an example of a suitable computing system environment100 on which the invention may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing environment100 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplaryoperating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-held,laptop or mobile computer or communications devices such as cell phonesand PDA's, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputers,mainframe computers, distributed computing environments that include anyof the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer in combination with hardware modules, includingcomponents of a microphone array 198. Generally, program modules includeroutines, programs, objects, components, data structures, etc., thatperform particular tasks or implement particular abstract data types.The invention may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices. With referenceto FIG. 1, an exemplary system for implementing the invention includes ageneral-purpose computing device in the form of a computer 110.

Components of computer 110 may include, but are not limited to, aprocessing unit 120, a system memory 130, and a system bus 121 thatcouples various system components including the system memory to theprocessing unit 120. The system bus 121 may be any of several types ofbus structures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. By wayof example, and not limitation, such architectures include IndustryStandard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus,Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA)local bus, and Peripheral Component Interconnect (PCI) bus also known asMezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes volatile andnonvolatile removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules, or other data.

Computer storage media includes, but is not limited to, RAM, ROM, PROM,EPROM, EEPROM, flash memory, or other memory technology; CD-ROM, digitalversatile disks (DVD), or other optical disk storage; magneticcassettes, magnetic tape, magnetic disk storage, or other magneticstorage devices; or any other medium which can be used to store thedesired information and which can be accessed by computer 110.Communication media typically embodies computer readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared, and other wireless media. Combinations of any ofthe above should also be included within the scope of computer readablemedia.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies. A user may enter commands andinformation into the computer 110 through input devices such as akeyboard 162 and pointing device 161, commonly referred to as a mouse,trackball, or touch pad.

Other input devices (not shown) may include a joystick, game pad,satellite dish, scanner, radio receiver, and a television or broadcastvideo receiver, or the like. These and other input devices are oftenconnected to the processing unit 120 through a wired or wireless userinput interface 160 that is coupled to the system bus 121, but may beconnected by other conventional interface and bus structures, such as,for example, a parallel port, a game port, a universal serial bus (USB),an IEEE 1394 interface, a Bluetooth™ wireless interface, an IEEE 802.11wireless interface, etc. Further, the computer 110 may also include aspeech or audio input device, such as a microphone or a microphone array198, as well as a loudspeaker 197 or other sound output device connectedvia an audio interface 199, again including conventional wired orwireless interfaces, such as, for example, parallel, serial, USB, IEEE1394, Bluetooth™, etc.

A monitor 191 or other type of display device is also connected to thesystem bus 121 via an interface, such as a video interface 190. Inaddition to the monitor, computers may also include other peripheraloutput devices such as a printer 196, which may be connected through anoutput peripheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device, or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks,intranets, and the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

The exemplary operating environment having now been discussed, theremaining part of this description will be devoted to a discussion ofthe program modules and processes embodying a “PeerStreamer” whichprovides dynamic real-time client control over a cluster of one or morepeers in a receiver-driven peer-to-peer (P2P) network for distributedmedia streaming.

2.0 Introduction:

A “PeerStreamer” as described herein provides receiver-drivenpeer-to-peer (P2P) media streaming for loosely coupled P2P networks.Peers in the network perform only simple operations, may cache all orpart of the streaming media, do not collaborate with other peers, may beunreliable, and may drop offline or come online during any givenstreaming session. Clients in the network operate in real-time tocoordinate peers, stream media from multiple peers, perform loadbalancing, handle online/offline states of peers, and perform decodingand rendering the streaming media.

Note that while the PeerStreamer system described herein is applicablefor use in large P2P networks with multiple clients and peers, thefollowing description will generally refer to individual clients forpurposes of clarity of explanation. Those skilled in the art willunderstand that the described system and method offered by thePeerStreamer is applicable to multiple clients. In addition, as thepeers described herein are used to serve the media to the receiver orclient, the cluster of peers in he P2P network are generally referred toherein either as peers, or as “serving peers.” It should also be notedthat these “serving peers” should not be confused with “media servers,”as described herein, from which particular streaming media filesinitially originate.

In general, the PeerStreamer operates in a P2P network such as thenetwork illustrated by FIG. 2. For a particular streaming session, a“server” 200 is defined as a node in the P2P network that initiallyoriginates the streaming media; a “client” (or receiver) 210 is definedas a node that currently requests the streaming media; and a “servingpeer” 220 is defined as a node that serves the client with a complete orpartial copy of the streaming media.

In general, the server 200, the client 210 and the serving peers 220 areall end-user nodes connected to a network such as the Internet. Becausethe server 200 is always capable of serving the streaming media, theserver node also acts as a serving peer 220. The server node 200 canalso perform media administrative functionalities that cannot beperformed by a serving peer 220, e.g., maintaining a list of availableserving peers, performing digital rights management (DRM) functionality,etc. In addition, as with conventional P2P schemes, the PeerStreamerdescribed herein benefits from increased efficiency as more and morestreaming peer nodes 220 are deployed. In particular, as the number ofstreaming peer nodes 220 increases, the load on the media server 200will decrease, thereby becoming less costly to run, while each clientnode 210 will be able to receive much better media quality during aparticular media streaming session.

In addition, it should be clear that as with many other P2P typenetworks, the role of particular nodes may change. For example, aparticular node may act as the client 210 in one particular streamingsession, while acting as a serving peer 220 in another session. Further,particular nodes can simultaneously act as both client nodes 210 andservers 200 or serving peers 220 to simultaneously stream one or moremedia files, or portions of media files, while receiving other streamingmedia from one or more other serving peers.

During a streaming session, the client 200 first locates a number ofclose-by peers 220 that hold some or all of the desired media, and thenstreams the media from the multiple peers (which may include the server200). Consequently, each serving peer 220 acts to assist the server 200by reducing the overall upload burden by servicing a portion of thedownload request of the client 210. As a result, the client 210,especially in the case where there are many clients, can often receivemuch better streaming media quality, as there is a significantly higherserving bandwidth available when there are many streaming peers 220 toassist the server 200.

As with any P2P network, each individual peer 220 does not directlybenefit from serving one or more clients 210. However, in oneembodiment, a conventional P2P “fairness mechanism” is used to ensurethat cooperating peers 220 receive higher priority in being served forsubsequent streaming requests than another peer that has not equallycooperated in acting as a serving peer. Consequently, when implementingsuch a fairness mechanism with the PeerStreamer, a cooperating peer 220can typically expect better media quality the next time it becomes aclient 210.

Consequently, recognizing the fact that each serving peer 220 iseffectively performing a favor for both the client 210 and the server200 during any particular streaming session, a good design philosophy isto ensure that the serving peer is lightweight and the P2P network isloosely coupled. In other words, the serving peer 220 should only needto perform very simple operations with low CPU load. Further, in oneembodiment, serving peers 220 can also elect to cache only part of themedia, so as to minimize the storage space that is essentially donatedby each serving peer. In addition, to reduce any bandwidth cost ofcommunications between peers 220, each serving peer should not berequired to collaborate with other peers. Finally, other programsrunning on any particular serving peer 220 may have a higher priority inclaiming the CPU and network resources at any particular point in time,or a particular peer may simply be turned on or off at any time. As aresult, particular serving peers 200 may be unreliable, with afluctuation in available serving bandwidth. In fact, particular servingpeers may simply drop offline, or come online, at any time during astreaming session.

Conversely, it is fair to increase the burden on the client 210 todevote resources to the streaming session. In particular, the client 210needs to receive the streaming media from multiple peers 220, so it isconnected to the peers already. Further, there is a motivation for theclient 210 to effectively coordinate or manage the peers 200 so as toimprove its own streaming experience. Consequently, the PeerStreamersystem and method described herein makes use of receiver-driven controlover the serving peer in a loosely coupled P2P network wherein theclient is responsible for sending and coordinating packet requests amongthe various streaming peers.

2.1 System Overview:

As noted above, the PeerStreamer described herein provides a system andmethod for receiver-driven peer-to-peer (P2P) media streaming forloosely coupled P2P networks. Peers in the network perform only simpleoperations, may cache all or part of the streaming media, do notcollaborate with other peers, may be unreliable, and may drop offline orcome online during any given streaming session. Clients (or receivers)in the network operate in real-time to coordinate peers, stream mediafrom multiple peers, perform load balancing, handle online/offlinestates of peers, and perform decoding and rendering the streaming media.

In general, the PeerStreamer provides receiver-driven media streaming.PeerStreamer operations begin with each receiving client retrieving alist of nearby serving peers that hold all or part of the requestedstreaming media. Note that in this context, a media server can also actas one of the serving peers. This list includes the IP addresses and thelistening ports of a set of one or more neighbor serving peers that holda complete or partial copy of the serving media. Methods for retrievingthis list include: 1) retrieving the list directly from the mediaserver; 2) retrieving the list from a known serving peer; and 3) using adistributed hash table (DHT) approach for identifying the serving peers.

Once the client has retrieved the list of available serving peers, theclient connects to each serving peer and obtains its “availabilityvector.” In general, the availability vector for each serving peer is acompact description of the exact portion of the media held by eachserving peer. This availability vector is then used by the client todetermine exactly what blocks of the encoded media are held by theserving peer.

For example, where a particular serving peer holds the entire servingmedia the availability vector of that peer can be a single flag thatindicates that the serving peer holds a complete media copy. Similarly,if the serving peer holds only a portion of the serving media, theavailability vector of that serving peer will signal to the client whatportion of the media is held by the serving peer, e.g., the number ofblocks of each packet and the block indexes that are held by the servingpeer.

Further, where additional coding is used, such as the erasure codingtechniques described below, the availability vector will include themedia erasure coding keys assigned to serving peer, and the number oferasure blocks held by the serving peer. In addition, if the servingpeer uses erasure coding and the media is also embedded coded, theavailability vector will include the assigned media erasure coding keys,the number of erasure blocks of each packet at the different bitratelevels used by the embedded coding.

Given the availability vector, the next step is for the client toretrieve the lengths of a “media header” and a “media structure” for themedia to be streamed from the peer cluster. After these lengths havebeen retrieved, the client calculates “data unit IDs” of the mediaheader and media structure, and retrieves them from one or more of thepeers in the peer cluster based on the knowing what peer has whatpackets IDs as a result of having analyzed the availability vector foreach serving peer.

Once the media header arrives, the client analyzes the media header, andthen configures or initializes whatever audio/video decoders andrendering devices that are needed for decoding and rendering or playingback the specific type of media being streamed (i.e., MPEG 1/2/4, WMA,WMV, etc.) Once this initial setup phase has been completed, the clientthen proceeds to coordinate ongoing streaming of the media body from thepeer cluster as described below. In particular, given the aforementionedmedia structure of the particular streaming media, the client calculatesdata unit IDs of packets of the streaming media, and then retrievesthose packets one by one from the various peers.

The client then periodically updates the serving peer list (using one ofthe aforementioned methods for identifying serving peers), and connectsto potential new serving peers. In a tested embodiment, the clientchecked for potential new serving peers by issuing periodic conventionalTCP connect function calls for each potential serving peer. After theclient establishes the connection to a new serving peer, it firstretrieves the aforementioned availability vector. The new peer may thenjoin the other active peers in the cluster, at the direction of thereceiver/client. The client then coordinates the peers, balances theserving load of the peers according to their serving bandwidths andcontent availability, and redirects unfulfilled requests of disconnectedor timed-out peers to one or more of the other active peers. Thestreaming operation then continues in this manner until the entirestreaming media is received, or the streaming operation is stopped bythe user.

2.2 System Architectural Overview:

The processes summarized above are illustrated by the general systemdiagram of FIG. 3. In particular, the system diagram of FIG. 3illustrates the interrelationships between program modules forimplementing a PeerStreamer, as described herein. It should be notedthat any boxes and interconnections between boxes that are representedby broken or dashed lines in FIG. 3 represent alternate embodiments ofthe PeerStreamer described herein, and that any or all of thesealternate embodiments, as described below, may be used in combinationwith other alternate embodiments that are described throughout thisdocument.

In general, the PeerStreamer begins operation with respect each client210 by having the client retrieve use a peer location module 305 toretrieve or identify a list 310 of nearby serving peers 220 that holdall or part of the requested streaming media. Note that in this context,the media server 200 can also act as one of the serving peers 220.Various methods are used by the peer location module 305 for retrievingthe peer list 310. For example, in one embodiment, the peer list 310 isprovided directly from the server 200. In another embodiment, the peerlist 310 is retrieved from a known serving peer 220. Finally, in yetanother embodiment, a conventional distributed hash table (DHT) is usedby the peer location module 305 to identify the serving peers 220. Asnoted above, the peer list 305 includes the IP addresses and thelistening ports of one or more neighboring serving peers 220 that hold acomplete or partial copy of the serving media.

The serving media itself is encoded by a media coding module 300existing on the server 200 using any of a number of conventional codecs,including for example, MPEG 1/2/4, WMA, WMV, etc. Note that the codecused to encode the media may be either embedded, or non-embedded, asdescribed in further detail herein. Further, in one embodiment, a“high-rate erasure resilient coding” as described in further detailbelow is used in combination with any of the codecs to provide forincreased robustness to inherently unreliable serving peers 220.

Initially, the encoded media exists only on the server on which thatmedia was originally encoded. It is then distributed, in whole or inpart to one or more of the serving peers 220 (again, the server 200 mayalso act as a serving peer for purposes of media streaming).Distribution to the serving peers 220 is the result of either directdistribution of packets of the media stream to the peers, or as a resultof having one or more of the peers that have already streamed that media(when acting as a client 210) simply cache all or part of the media asit is initially streamed to that serving peer. In any case, it isassumed for purposes of explanation that there are a number of knownpeers (as defined by the peer list 310), and that each peer holds all orpart of the encoded media to be streamed.

Once the client 210 has retrieved the list 310 of available servingpeers, the client connects to each serving peer 220 via an availabilityvector retrieval module 320 which retrieves the aforementionedavailability vector from each peer. Next, given the information of theavailability vector for each peer 320, the client 210 then uses a mediaheader/media structure analysis module 325 to retrieve the lengths of a“media header” and a “media structure” for the media header and themedia structure to be streamed from the peer cluster 220. After theselengths have been retrieved, the client 210 client analyzes the mediaheader, and then uses a client configuration module 330 to configure orinitialize whatever audio/video decoders and rendering devices that areneeded for decoding and rendering or playing back the specific type ofmedia being streamed (i.e., MPEG 1/2/4, WMA, WMV, etc.).

In addition, the media header/media structure analysis module 325 alsomakes a determination from an analysis of the media structure and mediaheader as to whether either or both embedded coded media or high-rateerasure resilient coding has been used in encoding the media to bestreamed.

A data unit ID calculation module 335 is then used to calculate “dataunit IDs” for packets of streaming media based on the informationincluded in the media header and media structure. A data unit requestmodule 340 then uses computed data unit IDs to request specific packetsor data blocks of the streaming media form various peers in the peercluster 220.

In the case where the PeerStreamer uses embedded coded media, thestreaming bitrates vary according to available serving bandwidths andclient queue status, as described in further detail below. In this case,ongoing requests for retrieval of media packets or data units by thedata unit request module 340 corresponds to those packets (or datablocks) that will provide the minimum rate distortion based on theavailable bandwidth. Further, in the additional case where high-rateerasure resilient coding is used, multiple serving peers hold partialmedia without conflict, such that clients simply retrieve fixed numbersof erasure coded blocks regardless of where and what specific blocks areretrieved.

In any case, as the client 210 retrieves streaming blocks of the mediavia a data unit processing module 345, the client will either pass thosepackets to be decoded, as described below, or the data unit processingmodule will first reconstruct the packets of the media stream from datablocks (see the discussion of high-rate erasure coding below). Inaddition, the client 210 will periodically update the serving peer list310 (using one of the aforementioned methods for identifying servingpeers). Whenever the list 310 is updated, or at some desired frequency,the client 210 will connect to potential new serving peers to retrievethe aforementioned availability vector. The new peer may then join theother active peers in the cluster 220, at the direction of thereceiver/client 210.

The client 210 then coordinates the peers 320, balances the serving loadof the peers according to their serving bandwidths and contentavailability, and redirects unfulfilled requests of disconnected ortimed-out peers to one or more of the other active peers. The streamingoperation then continues in this manner until the entire streaming mediais received and decoded rendered and played back via adecode/render/playback module 350. Note that playback of the decodedmedia is provided via conventional display devices 355 and/or speakers360 which are provided their input from the decode/render/playbackmodule 350.

3.0 Operation Overview:

The above-described program modules are employed for implementing thePeerStreamer. As summarized above, the PeerStreamer providesreceiver-driven peer-to-peer (P2P) media streaming for loosely coupledP2P networks. The following sections provide a detailed discussion ofthe operation of the PeerStreamer, and of exemplary methods forimplementing the program modules described in Section 2 with respect toFIG. 2. In particular, following the detailed description of thePeerStreamer operation provided below in Sections 3.1 and 3.2, anoperational flow diagram is presented in FIG. 10 which summarizes theoverall operation of the PeerStreamer in view of that detaileddescription.

3.1 Operational Details of the PeerStreamer:

The following paragraphs detail specific operational and alternateembodiments of the PeerStreamer described herein. In particular, thefollowing paragraphs describe a “streaming media model” used by thePeerStreamer; a “media structure” of the requested streaming media(basically a “companion file” which defines characteristics of thestreaming media needed to compute data ID's for retrieving media packetsor “data units”; PeerStreamer data units which represent fixed sizeportions of media packets for streaming; partial caching of media forreducing storage requirements; high-rate erasure coding of media forincreasing robustness of the PeerStreamer system to inherentlyunreliable serving peers.

3.1.1 Streaming Media Model:

In general, streaming media consists of a stream of packets that aredecoded and rendered as they arrive (hence the name streaming). Withoutstreaming, the entire media has to be downloaded in one big chunk beforeit can be used. The general structure of a streaming media file used bythe PeerStreamer is illustrated in FIG. 4.

In particular, as illustrated by FIG. 4, the media is led by a “mediaheader,” which contains global information describing the media, e.g.,the number of channels in the media, the properties and characteristics(audio sampling rate, video resolution/frame rate) of each channel,codecs used, author/copyright holder of the media, etc. The media headeris usually downloaded before the start of the streaming session, so thatthe client may set up the necessary tools to decode and render thesubsequently received packets. Note that streaming media may consist ofseveral channels, each of which is a separate media component that canbe independently selected and decoded, e.g., an English audio track, aSpanish audio track, a 4:3 video, a 16:9 video; etc.

The media header is followed by a sequence of media packets, each ofwhich contains the compressed bitstream of a certain channel spanningacross a short time period. Each media packet is led by a packet header,which contains information such as the channel index, the beginningtimestamp of the packet, the duration of the packet, as well as a numberof flags, e.g., whether the packet is a key frame (e.g., an MPEG Iframe), whether the packet is an embedded coded packet (with truncatablebitstream), etc. The compressed bitstream of the packet then follows.

Most of the conventional compressed media codecs today, such asMPEG1/2/4 audio/video, WMA/WMV, RealAudio®/RealVideo®, etc., generatenon-embedded coded media packets. Consequently, the size of the mediapackets generated by such systems can not be changed. Moreover, wheneverone of the media packets in such a bitstream is lost or overly delayed,the result is either that the media is not decodable, or the playbackbecomes choppy or intermittent, thereby degrading the playback qualityof the streaming media. In order to remain compatible with theseconventional codecs, in one embodiment, the PeerStreamer system andmethod allows media packets to be non-embedded coded (non-scalable).However, in addition to supporting traditional compressed media formats,the PeerStreamer also supports embedded coded media in one embodiment.

With embedded coded media, each media packet is encoded in such a waythat it can be independently truncated afterwards. In general, two typesof embedded coding are supported by the PeerStreamer, bitplane codingand enhancement layer encoding. Note that both types of embedded codingare well known to those skilled in the art. Consequently, such codingwill only be generally described in the following paragraphs.

For example, with bitplane coding, scalable coding of the media blocksis generally achieved by coding a block of audio/video transformcoefficients bitplane-by-bitplane, from the most significant bitplane(MSB) to the least significant bitplane (LSB). If the bitstream istruncated after encoding, the information is retained for several of themost significant bitplanes of all the coefficients. Moreover, thetruncated bitstream corresponds to a lower bitrate compressed bitstream,which can be considered as embedded in the higher bitrate compressedbitstream, hence the name embedded coding. As a result, the media packetgenerated by the embedded coder can be truncated, with a gracefulrate-distortion trade-off.

With enhancement layer coding, the media content is compressed into abase layer and one or more enhancement layers, each of which typicallyoccupies a separate channel. In particular, such coding allows a minimumquality media stream to be received be subscribing to the base layer.With the addition of each successive enhancement layer, the quality ofthe decoded media improves. Consequently, with such systems, thereceiver or client typically optimizes the quality of receivedinformation by subscribing to the base layer, and as many enhancementlayers as possible, depending upon available bandwidth.

3.1.2 PeerStreamer Media Structure:

To operate in a receiver-driven mode, the PeerStreamer client needs toknow the structure of the to-be-requested media packets, so that it mayknow what packets and what portion of each packet to request from eachpeer. This information is provided in a type of “companion file” whichincludes a definition of the structure of the streaming media to berequested. In general, this media structure provides the PeerStreamerclient with a bird's eye view of the entire media (such as, for example,the beginning timestamp of each packet, the duration of each packet,etc.), so that it can plan the P2P streaming session intelligently, andmake sure that particular media packets arrive in time for decoding andrendering. Note that the companion file containing the media structureinformation is initially generated at the time that the media file isoriginally encoded and is then streamed to the client upon request atthe start of each streaming session along with the initial request forthe media header. Note that the information in the companion file canalso be generated by analyzing the media header and packet headerinformation after the media has been encoded by a conventional codec.

In particular, the media structure of a set of packets is comprised ofthe packet headers plus the packet bitstream lengths. Consequently, thisinformation can be used by the client to determine which specificpackets should be requested, the time that those packets should berequested, and the peer from which those packets should be requested.Consequently, the PeerStreamer first retrieves the media structure ofthe entire media in a streaming “setup” stage. Retrieval of thisinformation prior to actually streaming the media causes a small delayin the startup of streaming. However, by retrieving this informationfirst, prior to media streaming, there is no additional cost inbandwidth (during media streaming) for serving the media structureinformation to the client.

Note that aforementioned delay in beginning streaming is typically verysmall relative to the overall length of the streaming media. Forexample, in a tested embodiment of the PeerStreamer, five test movieclips ranging from 31 megabytes (MB) to 49 MB in size had mediastructure companion files in the range of about 37 kilobytes (KB) toabout 53 KB. Therefore, the media structure size has been observed to beon the order of about 0.10–0.15% of the overall media body. Therefore,assuming that the serving bandwidths are greater than or equal to themedia bitrate, and the media structure is 0.15% of the media body,downloading the media structure of a 10 minute clip causes an additionaldelay of less than 0.9 s.

In a related embodiment, partial media structures are generated forsequential media segments of some predetermined length (i.e., 10seconds, 30 seconds, 1 minute, etc.). Each partial media structure isthen only retrieved before the corresponding media segment is to bestreamed in the near future. This slightly increases bandwidthrequirements since media structure requests and transmissions maycoexist with media packet requests and transmissions. However, since thesize of the media structure is so small in this case, the effect onoverall bandwidth requirements is typically negligible.

3.1.3 PeerStreamer Data Units:

In one embodiment, the PeerStreamer breaks the media packet, the mediaheader and the media structure into fixed size data units of length L.The reason for using fixed size data units is that the PeerStreamerclient and the serving peers can then pre-allocate memory blocks of sizeL, thus avoiding costly memory allocation operations during thestreaming process. Further, splitting the media packets (potentiallyvery large) into small fixed size data units also allows thePeerStreamer client to distribute the serving load to the peers with asmaller granularity, thereby achieving better bandwidth load balancingamong the peers.

In general, a splitting of a length P packet (which can be the mediapacket, the media header or the media structure) into blocks of size Lis achieved by splitting each packet into ┌P/L┐ data units, where ┌x┐ isa conventional ceiling function that returns the smallest integer thatis larger than or equal to x. All data units then have a fixed length L,except potentially the last data unit of each packet, which is of lengthP mod L.

In the case where non-embedded coding of the media is used, the dataunits comprising each media packet cannot be dropped during the networktransmission without loss of media playback quality. These data packetsare therefore are designated as “essential data units,” as they must allbe received.

Conversely, when an embedded coded media packet is split into dataunits, only the base layer data unit must be delivered, the remainingdata units may be optionally dropped if the serving bandwidths are notsufficient. These optional data units are designated as “non-essentialdata units.” The bandwidth required for the serving of the non-essentialdata units can be calculated as follows. For example, in the case ofembedded coding, a media packet will last T seconds. Assuming the mediapacket is split into a number of data units, in order to serve the dataunit at layer i to the client, all data units below layer i must also beserved to the client. As a result, the serving bandwidth required toserve the data unit at layer i is:R _(i)=(i+1)L/T   Equation 1

Therefore, Equation 1 provides the bitrate R of the data unit whenrespect to embedded coded media. The PeerStreamer client then adjusts tochanging serving bandwidths by dropping non-essential data units thatwould result in a bitrate above the available serving bandwidth.

In either case, whether the media is non-embedded coded or embeddedcoded, all data units of a particular media steam, including the dataunits of the media packet, the media header and the media structure, aremapped into a unique ID space. For example, in a tested embodiment, thedata units of the media packets were indexed from 0x00000000 to0xfdffffff (Hexadecimal), the data units of the media header from0xfe000000–0xfeffffff, and the data units of the media structure from0xff000000–0xffffffff. The data units used in this tested embodiment areof the PeerStreamer are illustrated in FIG. 5.

Note that to obtain the data unit IDs of the media header and the mediastructure, the lengths of the media header and the media structure arefirst needed. These are referred to as their “mega-structure.” To obtainthe data unit IDs of the media packets, the lengths of the media packetbitstream is needed. This information is included in the mediastructure.

3.1.4 Partial Caching of Media:

For serving purposes, each serving peer only needs to hold a portion ofthe media that is proportional to its serving bandwidth. Frequently, theserving (or upload bandwidth) of most computers connected to theInternet is substantially less than its download bandwidth (whichdictates the highest streaming bitrate that each particular node mayreceive). Consequently, each end-user node on the Internet tends to havean imbalance between its upload bandwidth and its download bandwidth.For example, given a node on a typical commercial ADSL/cable modemnetwork available to home users, it is not uncommon for the downloadbandwidth to be an order of magnitude higher than its upload bandwidth.Similarly, nodes on a campus/corporate network typically have cappedserving bandwidths so that the participation of any given node in P2Ptype activities will not affect other mission-critical functions.

Consequently, since each serving peer is not typically individuallycapable of serving an entire media stream to a client, there is no needto cache the entire media stream on any one serving peer. Therefore, aneffective way to decrease the amount of storage resources required byany of the serving peers is to allow each serving peer to hold only aportion of the media that is to be streamed. For example, if the bitrateneeded to stream non-embedded coded media is R, and the maximum servingbandwidth provided by a peer in a streaming session is B, each peer nodeonly needs to keep p portion of the streaming media in its cache, wherethe value p is denoted by Equation 2:p=max(1.0,B/R)   Equation 2

For example, assuming that the media bitrate is twice the servingbandwidth, i.e., R=2B. Then the serving peer only needs to keep half ofthe streaming media in its storage since that peer alone can not servethe client at the full streaming bitrate. In fact, given theaforementioned limitations of this example, the best that the peer cando is to supply at most half the media. Consequently, the peer onlyneeds to keep half of the media in its cache. The rest of the media tobe streamed must then be supplied by the other serving peers.

Further, it should then be noted that a combination of Equations 1 and 2then allows for a determination of the amount of amount of media to keepfor the case where embedded coded media is used. As discussed above inSection 3.1.3, the media packets of the embedded coded media are splitinto a number of data units with different bitrates. Therefore, With Rbeing the bitrate of the data unit for a particular layer L, Equation 2now gives the portion of the media to be kept for that data unit. Forexample, as illustrated by FIG. 6, an embedded media packet can be splitinto a plurality of data units (8 in this example). The amount of mediathat needs to be cached for each data unit (with L/T=0.5 B) is shownthen determined in accordance with Equation 2, as illustrated by in FIG.6.

However, in one embodiment, where a storage resources of a particularserving peer is sufficiently large, the serving peer may elect to cachea larger portion of the media by simply using a higher “potentialserving bandwidth,” B′, in Equation 2. The extra portion of the mediacached then enables the media to be served in a choppy, yet high qualityfashion. For example, assuming that each serving peer elects to use apotential serving bandwidth B′ of twice of its actual serving bandwidth,i.e., B′=2 B, the resultant amount of media in the P2P network will beenough for the client to retrieve the media at half the streaming rate.In other words, assuming that the aggregated serving bandwidths of allthe available peers are larger than R/2, the client should be able tofirst download half the media, then continuously stream and playback theremaining half. Similarly, the client can also elect to download aT_(s)/2 segment of the media (with time T_(s)), continuously streamanother T_(s)/2 segment and playback the segment, then download andstream another segment. The streaming media may thus be played back atrate R, albeit in a choppy fashion.

3.1.5 High-rate Erasure Coding of Media:

As noted above, peers may be inherently unreliable. Consequently, it isadvantageous to provide some means for providing increased redundancy inthe PeerStreamer system and method so as to effectively handle theinherently unreliable serving behavior of serving peers. Dealing withthis issue raises a number of concerns that must be addressed. Forexample, a determination of which portion p of the media should be keptby each peer is of concern. Further, since the media is ultimately splitinto the aforementioned data units, a determination of which portion pof the data units should each peer maintain is also of concern.

One strategy to address these issues is to simply separate each dataunit into k blocks. The peer keeping p portion of the media may thenrandomly hold ┌k·p┐ blocks, with ┌x┐ being the aforementioned ceilingfunction. However, one problem with the randomness of this scheme isthat even if there are many more than k blocks available in the peercluster, it is possible that the cluster as a whole may lack aparticular block j, thereby rendering the entire data unitirretrievable. Further, in such a scheme, the client is stillresponsible for locating each and every distinct block from the peers,which complicates the design of the protocol between the client and thepeers.

Consequently, a better strategy is to use a “high rate erasure resilientcode” to ensure that one or more of the peers will have the data blocksnecessary to reconstruct particular data units while simplifying thedemand on the client to identify which of the peers contains thenecessary data. In general, an erasure resilient code is a block errorcorrection code with parameters (n, k), where k is the number oforiginal messages, and n is the number of coded messages. High rateerasure resilient code satisfies the property that n is much larger thank, thus the k original messages are expanded into a much larger codedmessage space of n messages. While erasure coding techniques are ingeneral fairly well known for coding data, the application of suchtechniques for streaming media in a P2P network environment, asdescribed herein, are not known.

As a block error correction code, the operation of the high rate erasureresilient code can be described through a matrix multiplication over theGalois Field GF(p):

$\begin{matrix}{{\begin{bmatrix}c_{0} \\c_{1} \\\vdots \\\vdots \\c_{n - 1}\end{bmatrix} = {G\begin{bmatrix}x_{0} \\x_{1} \\\vdots \\x_{k - 1}\end{bmatrix}}},} & {{Equation}\mspace{14mu} 3}\end{matrix}$where p is the order of the Galois Field, {x₀, x₁, . . . , x_(k-1)} arethe original messages, {c₀, c₁, . . . , c_(n-1)} are the coded messages,and G is the generator matrix. Note that Equation 3 is not used togenerate all of the coded messages at once. Instead, the generatormatrix G defines a coded message space. Therefore, when the clientreceives k coded messages {c′₀, c′₁, . . . , c′_(k-1)}, they can berepresented by Equation 4 as:

$\begin{matrix}{{\begin{bmatrix}c_{0}^{\prime} \\c_{1}^{\prime} \\\vdots \\c_{k - 1}^{\prime}\end{bmatrix} = {G_{k}\begin{bmatrix}x_{0} \\x_{1} \\\vdots \\x_{k - 1}\end{bmatrix}}},} & {{Equation}\mspace{14mu} 4}\end{matrix}$where G_(k) is a sub-generator matrix formed by the k rows of thegenerator matrix G that correspond to the coded messages. Further, ifthe sub-generator matrix G_(k) has full rank k, then the matrix G_(k)can be inversed, and thus the original messages can be decoded.

There are several well known erasure coding technologies that may beused, including, for example, Reed-Solomon erasure codes, tornado codes,and LPDC codes. However, in one embodiment, the PeerStreamer provides anew high rate erasure resilient code based on a modified Reed-Solomoncode on the Galois Field GF(2¹⁶). In this example, the number of theoriginal messages k is 16. The size of the coded message space n is2¹⁶=65536. Reed-Solomon code is a maximum distance separable (MDS) code.Consequently, any 16 rows of the generator matrix G forms asub-generator matrix with full rank 16. In other words, the originalmessages can be recovered from any 16 coded messages. It should be notedthat other field sizes, p, may also be used, and that the PeerStreameris not limited to use of the particular field size described herein.Further, for embodiments using non-MDS erasure coding, it may benecessary to retrieve k′≧k blocks to recover the original message,depending upon the particular erasure coding used. The Reed-Solomonbased erasure codes were used, in part, because they are MDS codes, andthey can be efficiently encoded and decoded while placing only a smallcomputational overhead onto the CPU of most conventional computers.

With a high rate (n, k) erasure resilient code, each peer node isassigned k keys in the coded message space of n, with each key being therow index of the generator matrix G. The key assignment may be carriedout by the server. Further, if the number of peers caching the media issmaller than n/k, it is possible to assign each peer a unique set ofkeys. As a result, it can be guaranteed that each peer holds distinctivecoded messages. While this strategy provides a number of benefits, itstill requires a central coordination node (such as the server).

Consequently, in another embodiment, the role of the centralcoordination node is eliminated by allowing each peer to choose k randomkeys. If the number of peer nodes is greater than n/k or the key isassigned with no central coordination node, certain peer nodes may holdthe same keys. Nevertheless, in most media streaming sessions where theclient is connected to m peers, m is usually much smaller than n/k.Therefore, the probability that two serving peers happen to hold thesame key, and thus that one key of one of the peers is not useful, issmall. However, even if there is key conflict, the client can easilyidentify such conflicts when it first connects to the peers. In the casewhere such a conflict is identified, the client simply invalidates oneof the duplicated keys for the remainder of the streaming session.Consequently, the client does not need to actually address the keyconflict during the streaming process.

For example, assume that S1 and S2 are the erasure coded key spaces ofserving peer 1 and serving peer 2, respectively, and that S1={1, 7, 23,43, 48} and S2={3, 7, 28, 49, 99}. Clearly, key space S1 and S2 aredifferent. However, key 7 is shared by the two key spaces, therefore,serving peer 1 and serving peer 2 may hold an erasure coded blocksharing the same key, i.e., key “7”. Therefore, prior to requestingparticular coded blocks, key “7” is invalidated with respect to one ofthe serving peers so that the block coded by key “7” is retrieved fromonly one the peers, thereby avoiding any decoding conflicts caused byduplicate keys. However, it should be noted that in the case where oneserving peers goes offline during media streaming operations, particularinvalidated coding keys of another serving peer may be revalidated wherethe offline serving peer was previously in conflict as a result of usingone or more duplicate keys.

With (65536, 16) Reed-Solomon code, each data unit is dissected into 16blocks. Using a set of pre-assigned keys, the peer chooses to cache ┌16p┐ erasure encoded blocks, where p is a parameter calculated fromEquations 1 and 2. The keys assigned to the peer, and its maximumserving bandwidth B, constitute the aforementioned availability vectorof the peer, as the client can determine how many and what erasure codedblocks (by data unit/block ID) are held by the peer by using theinformation provided by that peers availability vector. Again, theclient resolves any key conflicts at the time that each peer isinitially connected. During the streaming session, the client can thenretrieve any k coded messages from any serving peer nodes, and decodethe associated data unit.

Further, it is not necessary to store an entire set of the coded blocksfor decoding particular data units on any one serving peer. In otherwords, the number of blocks held by any particular serving peer for anyparticular data unit may be less than k. Therefore, rather than wastecomputing power to compute every coded block for every coding key, inone embodiment, only those coded blocks that are actually beingdelivered to specific peers are generated. In other words, where j<kblocks are stored on a particular serving peer, only j blocks should begenerated for the particular data unit.

3.2 Implementation of PeerStreamer Operations in a P2P Network:

Implementation of the PeerStreamer operations is described in thefollowing paragraphs in view of the preceding discussion of theoperational details of the PeerStreamer. In particular, the followingparagraphs describe the location of serving peers by the client; setupof client decoding and rendering based on the retrieved media structure;PeerStreamer network connections; streaming bitrate control;PeerStreamer client requests and peer replies; and finally, PeerStreamerrequest and staging queues.

3.2.1 Locating Serving Peers:

As noted above, the first task performed by the client is to obtain theIP addresses and the listening ports of a list of neighboring servingpeers that hold a complete or partial copy of the serving media.Further, this list is also updated during the media streaming session.As explained above, general approaches for obtaining this listinclude: 1) retrieving the list from the server; 2) retrieving the listfrom a known serving peer; and 3) using a distributed hash table (DHT)approach for identifying serving peers where neither the media servernor a serving peer is known in advance.

3.2.2 Decoding and Rendering Setup:

After securing the serving peer list, the client attempts to connect toeach of the serving peers. Once connected, the client retrieves theavailability vector of each peer, and resolves any key conflicts, asdescribed above. Then, the client retrieves the lengths of the mediaheader and the media structure from one of the peers. After both lengthsare retrieved, the IDs of the data units of the media header and mediastructure are constructed. The media header and the media structure canthen be retrieved in a P2P fashion as described in further detail inSection 3.2.6. Once the media header is retrieved, the client determineswhich decoders and renderers should be initialized to decode and renderthe media as it is streamed to the client.

In a tested embodiment implemented using DirectX™, this setup wasaccomplished by first constructing a DirectShow™ filter graph from theinformation provided in the media header. It should be noted that thePeerStreamer described herein is not limited to implementation usingDirectX™ functionality, and that the use of DirectX™, and its discussionrelative to a tested embodiment, is provided for purposes of explanationonly for describing setup of the client computer in decoding renderingthe streaming media for client playback.

Therefore, assuming a DirectX™ implementation for client setup, thenetwork component of the client is represented by a DirectShow™ networksource filter, whose output is fed into the proper audio/video decoderDirectX™ media object (DMO). This DMO is then further connected to theappropriate audio/video rendering device. For example, a sampleDirectShow™ filter graph of a clients PeerStreamer media streamingsession is illustrated by FIG. 7. In this example, the streamed media isnon-embedded coded. The audio bitstream is compressed by WMA, and thevideo bitstream is compressed by MPEG-4.

One advantage of using implementing the PeerStreamer client setup viathe DirectShow™ framework is that it may use a huge library of existingaudio/video encoders/decoders developed under DirectShow™. For example,with DirectShow™, the PeerStreamer client is capable of decoding andrendering media coded by a variety of codecs, including, for example,MPEG 1/2/4, WMANWMV, Indeo Video, etc., or any other codec that has aDirectShow™ decoder DMO component. DirectShow™ also provides additionalaudio/video processing modules, such as resolution/color spaceconversion and de-interlacing, so that the decoded audio/video may beautomatically matched to the capabilities of the client's audio/videorendering devices.

Further, DirectShow™ automatically handles synchronization of theaudio/video tracks. For example, where the audio stream holds areference clock of the entire stream, when playing a streaming video,DirectShow™ ensures that the system timing clock of the video streamstays as close as possible to the clock of the audio stream foraddressing issues such as lip sync. Finally, DirectShow applications areinherently multithreaded. Consequently, on a multiprocessor PC (or onewith Hyper-Threading enabled), the computation load of variouscomponents of the client, e.g., the network component, the audiodecoder, the video decoder, and the audio/video rendering engine, etc.,can be distributed onto the multiple processors. This greatly speeds upthe execution of the client, and allows more complex audio/videodecoders to be used.

Finally, it should again be noted that the PeerStreamer described hereinis not limited to implementation using DirectX™ functionality, and thatthe use of DirectX™, and its discussion relative to a tested embodimentis provided for purposes of explanation only for describing setup of theclient computer in decoding rendering the streaming media for clientplayback.

3.2.3 PeerStreamer Network Link and Packet Loss Management:

Most media streaming clients, such as, for example, Windows® mediaplayer or RealPlayer®, use the well known real time transport protocol(RTP), which is carried on top of UDP. The UDP/RTP protocol is typicallychosen for media streaming applications because: 1) the UDP protocolsupports IP multicast, which can be efficient in sending media to a setof nodes on an IP multicast enabled network; and 2) the UDP protocoldoes not have any re-transmission or data-rate management functionality.Consequently, the streaming server and client may implement advancedpacket delivery functionality, e.g., forward error correction (FEC), toensure the timely delivery of media packets.

However, in contrast to the well known media streaming schemesidentified above, the PeerStreamer uses TCP connections as the networklinks between the client and the serving peers. One reason for choosingTCP connections rather than conventional UDP/RTP protocols is that IPmulticast is not widely deployed in the real world because of issuessuch as inter-domain routing protocols, ISP business models (chargingmodels), congestion control along the distribution tree and so forth.

In addition, like many commercial media players, the PeerStreamer clientincorporates a streaming media buffer (of 4 s in a tested embodiment) tocombat network anomalies such as jitter and congestion. In fact, given astreaming media buffer many times larger than the round trip time (RTT)between the client and the serving peer, the TCP ARQ (automated repeatedrequest) mechanism is good enough for the delivery of the media packetsin sufficient time to provide smooth playback of the streaming media.

In general, there are three well known mechanisms (with a large numberof well known variations) for addressing media packet loss. For example,these mechanisms generally include: FEC, selective packetretransmission, and automatic repeat request (ARQ). Any of these packetloss mechanisms can be used by the PeerStreamer. However, as explainedbelow, there are advantages to using particular mechanisms over others.

In particular, for the Internet channel, which can be considered as anerasure channel with changing characteristics and an unknown packet lossratio, a fixed FEC scheme either wastes bandwidth (with too muchprotection) or fails to recover the lost packets (with too littleprotection). It thus does not efficiently utilize the bandwidth resourcebetween the client and the peer. Therefore, with a streaming buffer manytimes larger than the RTT, and thus plenty of chances forretransmission, retransmission based error protection (such as selectiveretransmission and ARQ) is preferable over FEC.

Considering ARQ and selective retransmission, it can be seen that in theInternet channel using the TCP protocol, selective retransmission willhave an edge over ARQ only if many packets are not selected to beretransmitted. For non-embedded coded media, a lost packet usually leadsto serious playback degradation, including failure to decode and provideplayback of particular packets. Therefore, the lost packet is almostalways retransmitted. Conversely, with embedded coded media, a lostpacket may not prevent the media from playing back. However, the loss ofa random packet still causes a number of derivative packets to be notuseable. As a result, only the topmost enhancement layer packets may notbe selected to be retransmitted.

In comparison to selective retransmission, ARQ always retransmits thepackets once they are requested; even they belong to the top mostenhancement layer. Nevertheless, the ARQ scheme can choose not torequest the top most enhancement layer packets of the following mediapackets, thus achieving the same bandwidth usage and perceived mediaplayback quality with the selective transmission scheme. Consequently,unless the network condition varies very quickly, the ARQ mechanismemployed by the TCP protocol is sufficient to handle the packet loss inmedia streaming.

Using TCP as the network protocol also provides several additionalbenefits over conventional media streaming schemes such as thoseidentified above. For example, with TCP, there is no need to dealexplicitly with flow control, throughput estimation, congestion controland avoidance, keep alive, etc. All of these issues are handledautomatically by the TCP protocol. The TCP protocol can also detect apeer going offline, and gracefully handle the shutdown of the connectionlink between the peer and the client.

3.2.4 PeerStreamer Streaming Bitrate Control with Embedded Coding:

Non-embedded coded media is preferably always streamed at the bitrate ofthe media to avoid degradation of media playback at the client. However,the streaming bitrate of embedded coded media may vary during thestreaming session.

Therefore, in one embodiment, the streaming bitrate R_(recv) for eachembedded coded media packet is first calculated by Equation 5, 6 and 7,as follows:R _(raw) =Th·(1+T _(rft) −T _(staging))+B _(staging) −B _(outstanding)  Equation 5R _(filter)=(1−α)R _(filter) +αR _(raw)   Equation 6R _(recv)=min(R _(min) , R _(inst))   Equation 7where Th is the aggregated serving bandwidths of the plurality ofserving, peers, T_(staging) is a target staging buffer size (with adefault of 2.5 s in a tested embodiment), T_(rft) is a desired requestfulfillment time (with a default of 1.0 s in a tested embodiment),B_(staging) is the length of the received packets in the staging queue,B_(outstanding) is the length of outstanding replies to be received,R_(min) is the base layer bitrate (with only essential data units), andα is a low pass control parameter.

The results of Equations 5–7 are then used to control the streamingbitrate R_(recv) by following the aggregated serving bandwidth Th andthe staging and request queue statuses, which are described in furtherdetail below in Section 3.2.6. Once the streaming bitrate is determined,the client only issues requests for the data units with a bitrate belowthe streaming bitrate R_(recv).

In a related embodiment, a more advanced strategy is used to control thebitrate R_(recv) by considering a distortion contribution of the dataunits as well. However, this requires that the client gains access tothe distortion (or the rate-distortion slope) of the data units, whichmust be included in the media structure and sent to the client. However,unlike existing information in the media structure, the distortion ofthe data units is not needed in decoding and is thus considered to beadditional overhead. Consequently, it is thus a trade-off between theamount of overhead to be sent to the client versus the rate-controlaccuracy.

3.2.5 PeerStreamer Data Block Requests and Replies:

The life of a client data block request and its reply be the peer isgenerally illustrated by FIG. 8. In particular, as illustrated by FIG.8, the client generates the request and sends it through the outboundTCP connection to a particular serving peer. Further, in networkdelivery, TCP may bundle the request with prior requests issued to thesame peer. If a prior request is lost in transmission, TCP handles theretransmission of the request as well.

After the packet request is delivered to the peer, it is stored in theTCP receiving buffer of the serving peer. The peer then processes therequests, one at a time. For each request, the peer reads the requestedblocks (which may or may not be erasure coded, depending upon the codingused) from its disk or memory storage, and sends the requested contentback to the client. In case the TCP socket from the serving peer to theclient is blocked, i.e., no more bandwidth is available, the servingpeer will block further client requests until the TCP connection opensup.

The interval between the time that the request is issued by the clientand its reply is received by the client is defined as the requestfulfillment time (RFT). The request is usually much smaller than itsreply, and the operations involved in processing the request, e.g., diskread, are typically trivial compared with the network delivery time usedto send the content back. Therefore, the RFT of the request, T′_(rft),is computed by Equation 8, as follows:T′ _(rft)=(B _(i, outstanding) +B _(cur))/Th _(i)   Equation 8where Th_(i) is the serving bandwidth of peer i, B_(i,outstanding) isthe length of unreceived replies before the request, and B_(cur) is thelength of the content requested. Therefore, RFT is determined as afunction of the serving bandwidth of the peer, the size of the requestand size of the unreceived content from the peer.

Once the requested content packet arrives at the client, it isimmediately moved to a staging queue. In the staging queue, the datablocks (which may include erasure coded blocks) from multiple peers arecombined and decoded into the data units, which are further combinedinto the media packet. Periodically, the client removes the deliveredmedia packets from the staging queue, and pushes them into thecorresponding audio/video decoder. After the media packets aredecompressed by the decoder, the uncompressed audio/video data streamsare sent to the audio/video rendering unit for streaming playback on theclient playback devices (display monitor, speakers, etc.).

In one embodiment, the buffers illustrated in FIG. 8 are used to combatnetwork anomalies such as the packet loss and jitter. (However, whenusing a DirectShow™ implementation, the uncompressed audio/videobuffers, are under the control of the DirectShow filter graph and arenot programmable). In a tested embodiment of the PeerStreamer, the sizeof the staging buffer was set to T_(staging)=2.5 s, the desired RFT wasset to T_(rft)=1.0 s, and the compressed audio/video buffer was set to0.5 s. Consequently, in this tested embodiment, the total buffer of thePeerStreamer client is thus around 4 s.

In the embodiment where erasure coding is used, each data block requestis formulated as the request of a group of erasure coded blocks of acertain data unit. The erasure coded block group is identifiable withthe start block index and the number of blocks requested. The data unitis identifiable through a 32 bit ID. The request is thus in the form of:Data_Unit_ID[32], Start_Index[4], Number_of_Blocks [4]  Equation 9where the number in the bracket is the number of bits of each component.

Therefore, as illustrated by Equation 9, in the case of erasure codedblocks, each request is 5 byte long. On the other hand, the contentrequested ranges in size from 128 to 2048 bytes (data unit lengthL=2048, k=16). As a result, the size of the request is only about 0.24%to 3.91% of the reply. Therefore the amount of the upload bandwidthspent by the client to send the request is thus very small relative tothe content requested.

3.2.6 PeerStreamer Request and Staging Queues:

As noted above, the PeerStreamer client maintains a single staging queueto hold received data blocks (which may be erasure coded) and from whichthe data blocks are assembled into the data units and then into themedia packets. The client also maintains a separate request queue foreach of the serving peers to hold the unfulfilled requests sent to eachpeer. One example of these request and staging queues is illustrated byFIG. 9.

The staging queue is the main streaming buffer of the PeerStreamerclient. All received contents are first deposited into the stagingqueue. The request queues serve three purposes: 1) to perform throughputcontrol and load balancing; 2) to identify the reply sent back by eachserving peer; and 3) to handle disconnected peers.

The first functionality of the request queue is to balance the loadamong the serving peers. In the case where the media is erasure coded,the request for a data unit is broken into the requests of multiplegroups of erasure coded blocks, with each group directed to one peer.The requests are generated through the following operations. Uponrequesting a data unit, the client first checks the availability vectorof the peers, and calculates the number of erasure coded blocks (a_(i))held by each peer for the data unit. If the total number of blocks heldby all peers online is less than k, the data unit is irretrievable. Ifthe irretrievable data unit is non-essential (i.e., non-base layer ofembedded coded media), the client simply skips the data unit.

Conversely, if the irretrievable data unit is essential, i.e., belongsto a non-embedded coded media packet or the base layer of an embeddedcoded media packet, the client cannot proceed with download and playbackof the streaming media. Therefore, in one embodiment it will wait formore peers to come online to supply the missing blocks. In an alternateembodiment, the client will skip the entire media packet, and mark it asmissing to the following audio/video decoder. The result will be a gapor skip in the rendered media. However, if one essential data unit isirretrievable from the peer cluster, it is very likely that morefollowing essential data units will also be irretrievable. Consequently,it is typically better to let the client wait until the data isavailable so as to provide a better playback experience to the user.

After ensuring that a particular data unit is retrievable, i.e.,Σ_(i)a_(i)≧k   Equation 10The client checks the space available in the request queue of each peer.It is desirable to maintain the RFT of each peer to be around a systemconstant T_(rft). In a tested embodiment, T_(rft) on the order of about1.0 s provided good results. (Note that using a too short request queuemay not effectively utilize the bandwidth from the client to the peer.)

In particular, in the event that the request packet sent by the clientis lost or delayed, the serving peer may be left with nothing to send,which wastes its serving bandwidth. Conversely, using an overly longrequest queue may prevent the client from quickly adapting to changes,such as the disconnection of one of the peers. Further, with the requestqueues for all peers being the same length in RFT, the capacity of therequest queue becomes proportional to its serving bandwidth:Th_(i)·T_(rft).

For example, assuming T_(rft) is 1.0 s, a peer with serving bandwidth of16 kbps allows 2 KB of unfulfilled requests pending in its requestqueue, while a peer with serving bandwidth 1 Mbps allows 128 KB ofunfulfilled requests pending. The number of erasure coded blocks thatcan be requested from a particular peer is thus capped by the space leftin its request queue:e _(i)=min(a _(i), (Th _(i) ·T _(rft) −B _(i,outstanding))/bk)  Equation 11where e_(i) is the number of erasure coded blocks that can be requestedfrom the peer i, and bk is the size of the erasure coded blocks.

Equation 11 guarantees that the client never sends out a request thathas an expected RFT greater than T_(rft). If the client cannot findenough current available erasure coded blocks, i.e.,Σ_(i)e_(i)<k   Equation 12it will wait until the request queue of the serving peer clears up. Thedata unit requests are only formed and sent to the peers whenΣ_(i)e_(i)≧k. The actual number of blocks (b_(i)) requested from acertain peer is calculated by:

$\begin{matrix}\left\{ \begin{matrix}{{{\sum\limits_{i}b_{i}} = k},} \\{{b_{i} = {\min\left( {e_{i},{c \cdot {Th}_{i}}} \right)}},}\end{matrix} \right. & {{Equation}\mspace{14mu} 13}\end{matrix}$where c is a constant that satisfies Σ₁b_(i)=k.

In general, the procedure outlined above allocates the serving load toeach peer in proportion to its serving bandwidth Th_(i) (Equation 13).It also ensures that the client does not request more blocks than from aparticular serving peer than are actually cached or stored by thatserving peer. Finally, this procedure also ensures that the RFT of therequest does not exceed T_(rft), as illustrated by Equation 11.

The second functionality of the request queue is to identify the contentsent back by each serving peer. As noted above, the PeerStreamer clientand peers communicate through TCP, which preserves the order of datatransmission, and guarantees packet delivery. Furthermore, each peerprocesses incoming requests in sequence. As a result, there is no needto specifically identify the content sent back, as it must be for thefirst request pending in the request queue for each peer.

With respect to the third functionality of the request queue notedabove, the request queue is also used to redirect the requests of thedisconnected peers. For example, whenever a particular serving peer isdisconnected from the client, the disconnection event is picked up bythe TCP protocol which then reports this disconnection to the client.The client then dynamically reassigns all unfulfilled requests pendingin the queue of the disconnected peer to one or more of the remainingpeers. The procedure for reassigning the request is very similar to theprocedure of assigning the request in the first place. The onlyexception is that the number of blocks already requested from thedisconnected peer must be considered in the request reassignment.

Finally, whenever erasure coded blocks arrive at the client, they areimmediately pulled away from the TCP socket. After pairing the arrivingcontent with the pending request, the fulfilled request is removed fromthe request queue. The identified erasure coded blocks are thendeposited into the staging queue. The size of the staging queueincreases as a result. If the staging queue reaches a predetermined sizeT_(staging), no further requests of the media packets/data units aresent. Once all erasure coded blocks of a certain data unit have beenreceived, the data unit is erasure decoded, and is marked as ready. Amedia packet becomes ready if all its requested data units are ready.Periodically, the audio/video decoder removes the “ready” media packetfrom the staging queue. This reduces the size of the staging queue, andmay trigger the generation of new media packet requests.

The media streaming operations described above then continue untilplayback of the media file is completed, or until such time as there areinsufficient peers available to stream the media, or a user terminatesthe streaming session.

3.3 PeerStreamer Operation:

The processes described above with respect to FIG. 2 through FIG. 9 areillustrated by the general operational flow diagram of FIG. 10. Ingeneral, FIG. 10 illustrates an exemplary operational flow diagramshowing several operational embodiments of the PeerStreamer. It shouldbe noted that any boxes and interconnections between boxes that arerepresented by broken or dashed lines in FIG. 10 represent alternateembodiments of the PeerStreamer described herein, and that any or all ofthese alternate embodiments, as described below, may be used incombination with other alternate embodiments that are describedthroughout this document.

In particular, as illustrated by FIG. 10, prior to media streamingoperations, the server 200 (which may also be one of the peers 220)encodes 1000 the media to be streamed. As described above, thePeerStreamer is capable of operating with any of a number ofconventional codecs, such as, for example, MPEG 1/2/4, WMA, WMV, etc. Inaddition, during the encoding process 1000, the server 200 alsogenerates both the aforementioned the media header, and the companionfile containing the media structure.

As described above, in one embodiment, once the media is encoded 1000,the encoded media packets are split 1005 into a number of data units ofa fixed size. Further, as with the encoded media, the media header andthe media structure are also split 1005 into a number of data units ofthe same fixed size as used to split the encoded media packets. Asexplained above, splitting 1005 this information into fixed length dataunits allows for both the client and the serving peers to pre-allocatememory blocks prior to media streaming operations, thereby avoidingcomputationally expensive memory allocation operations during thestreaming process. Further, the use of smaller data units allows forfiner control by the client over the exact amount of bandwidth expendedby each serving peer to meet client data unit requests during streamingoperations.

In addition to splitting 1005 the encoded media, the media header, andthe media structure into smaller data units, in one embodiment, anadditional layer of coding is used to provide increased redundancy in atypical P2P environment where serving peers are inherently unreliable.In particular, as described above, in one embodiment, the data units arefurther divided into a number of data blocks using a key-based high rateerasure resilient coding process 1010.

The use of such coding 1010 ensures that one or more of the peers willhave the data blocks necessary to reconstruct particular data unitswhile simplifying the demand on the client to identify which of thepeers contains the necessary data. Further, as noted above, in oneembodiment, the erasure resilient coding keys used by each serving peer220 are automatically assigned to each peer by the server 200. However,in another embodiment, each serving peer 220 simply chooses an erasureresilient coding key at random. These keys are then included along withthe aforementioned availability vector that is retrieved by the client210 when each peer 220 is initially contacted by the client. In therandom key embodiment, the client then invalidates the keys of one ormore peers where there is a key conflict for a given data unit.

Once the media has been initially encoded 1000, split into data units1005, and possibly further erasure coded 1010, the resulting data unitsor data blocks are then distributed 1015 to the various serving peers220. This distribution 1015 can be deliberate in the sense that theblocks or packets of the encoded media are simply provided in whole orin part to a number of peers where it is then cached or stored forfuture streaming operations when called by a client joined to the P2Pnetwork.

Alternately, as discussed above, whenever a client 210 streams aparticular media file, the recovered media packet is just the mediapacket after encoding operation 1000. They may be split into data units1005, and possibly further erasure coded 1010, and the client maymaintain at least a portion of the content that was streamed to it,possibly within local memory or storage. The client is then identifiedas a serving peer 220 (in the aforementioned peer list 310) for futurestreaming operations. One advantage to this embodiment is that while thenumber of peers containing portions of a particular media file isinitially low, thereby increasing demands on the server itself to meetserving requests, as time passes and more clients stream the media,those client will then be able act as peers for later streamingrequests. Consequently, there is no need to explicitly select servingpeers 220 to hold an initial cache of all or part of the media to bestreamed. As a result, any demands on the server are further lessenedwith respect to trying identifying peers willing to accept an initialcache of media to be streamed.

In either case, once the media has been distributed 1015 to the servingpeers 220, the client 210 then is ready to begin streaming requests tothose serving peers. Further, as noted above, the server 200 can alsoact as a serving peer 220 for the purposes of streaming to the client210. Again, in view of the above discussion, it should be clear thatwhile initial streaming of a particular media file may require greaterserver 200 involvement, as time passes, and more clients 210 stream thatmedia (and are then available to act as serving peers), the demands onthe server to actually act as a serving peer are reduced, or eveneliminated.

At this point, the client 210 begins a streaming session by firstretrieving the list 310 of available serving peers 220. As describedabove, this list 310 is retrieved directly from the server 200, from oneof the peers 220, or by using a conventional DHT method 315 foridentifying potential serving peers. Once the client 210 has retrievedthe peer list 310, the client then connects to each serving peer 220 andretrieves 1025 the availability vector from each peer. Further, in oneembodiment, the client 210 periodically checks for updates 1030 to thepeer list 310 during ongoing streaming operations. One advantage ofperforming such periodic checks 1030 is that in a large P2P network, itis probable that multiple serving peers are coming online and goingoffline at any given point in time. Consequently, ensuring that theclient 210 has an updated peer list 310 will allow the client to respondto the loss or degradation of a peer 220 that is currently streamingmedia to the client. Whenever a periodic check 1030 of the list 310indicates the addition of a new peer 220 to the list, the client 210again connects to the new peer and retrieves 1025 that new peer'savailability vector.

Once the client 210 has retrieved 1025 the availability vector of eachpeer 220, the client then retrieves 1035 the media header and mediastructure of the media to be streamed from one or more of the servingpeers by requesting data units corresponding that information from oneor more of the peers via a network connection between the client andthose peers.

As noted above, the media header generally contains global informationdescribing the media, e.g., the number of channels in the media, theproperties and characteristics (audio sampling rate, videoresolution/frame rate) of each channel, codecs used, author/copyrightholder of the media, etc. Consequently, retrieval of the media header atthe start of the media streaming session allows the client 220 to set upor initialize 1040 the necessary tools to decode 1070 and render 1075the subsequently received packets prior to receipt of those packetsduring the streaming session.

Further, after retrieving 1035 the media structure of the particularstreaming media, the client analyzes that media structure and calculatesdata unit IDs 1045 of data units of the streaming media that will needto be requested during the streaming process. The client 210 thenrequests those data units 1050, one by one, from one or more of theserving peers 220.

Further, as noted above, in the embodiment where erasure coding is usedin combination with random peer selection of coding keys, the client 210will invalidate duplicate keys on one or more of the peers 220 so as tomanage key conflicts 1055. In a related embodiment, the PeerStreameruses embedded coded media, and the data requests (and streamingbitrates) for each peer 220 are then managed 1060 according to availableserving bandwidths and the client 210 queue status. In this case,ongoing requests for data units 1050 correspond to those packets thatwill provide the minimum rate distortion based on the availablebandwidth of the various serving peers. In either case, as noted above,missing or late data units are again requested 1050 from either the sameor an alternate peer 220, depending upon whether embedded ornon-embedded coding has been used, the connection status of the peers,and the time remaining to request and receive the missing or late dataunit.

Finally, once all of the data units constituting a particular mediapacket have been retrieved in accordance with the client 220 request1050, those data packets are reassembled 1065 into the original mediapacket. Reassembled media packets are then decoded 1070, rendered 1075,and provided for playback on either or both a conventional displaydevice 355 or speakers 260.

The foregoing description of the PeerStreamer has been presented for thepurposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise form disclosed. Manymodifications and variations are possible in light of the aboveteaching. Further, it should be noted that any or all of theaforementioned alternate embodiments may be used in any combinationdesired to form additional hybrid embodiments of the PeerStreamer. It isintended that the scope of the invention be limited not by this detaileddescription, but rather by the claims appended hereto.

1. A computer-readable medium having computer executable instructionsfor providing client-driven streaming of multimedia data packets in apeer-to-peer (P2P) network, said computer executable instructionscomprising: maintaining a plurality of client request queues on a clientcomputer, each client request queue corresponding to one of a pluralityof serving peers in a cluster of serving peers; sending a client requestof one or more data packets from the client computer to one or more ofthe serving peers; wherein the client data packet requests to anyserving peer are provided via a reliable and order preserving link tothe serving peer, such that the serving peers do not need to identifythe data packets sent in reply to the client requests; adding each datapacket request to the corresponding request queue when it is sent fromthe client to one of the serving peers, removing each packet requestfrom the corresponding request queue when the corresponding data packetis received by the client from the serving peer; providing each receiveddata packet to a common staging queue maintained by the client; andassembling the data packets in the common staging queue intocorresponding multimedia data packets.
 2. The computer-readable mediumof claim 1 wherein maintaining a plurality of client request queues on aclient computer further comprises creating and maintaining additionalclient request queues corresponding to any additional serving peerswhich subsequently join the cluster of serving peers.
 3. Thecomputer-readable medium of claim 1 wherein maintaining a plurality ofclient request queues on a client computer further comprises moving anydata packet requests remaining in a request queue corresponding to aserving peer dropped from the cluster of serving peers to one or more ofthe other client request queues.
 4. The computer-readable medium ofclaim 1 wherein load balancing is maintained across each of the servingpeers in the cluster of serving peers by providing client management ofdata packet requests for maintaining an approximately consistent requestfulfillment time (RFT) across each of the serving peers in the clusterof serving peers.
 5. The computer-readable medium of claim 1 wherein thereliable and order preserving link to the serving peer is a TCPcommunications protocol.
 6. The computer-readable medium of claim 1wherein any incoming data packet received by the client from a servingpeer corresponds to a first data packet request in the request queuecorresponding to the serving peer from which the data packet wasreceived.
 7. The computer-readable medium of claim 1 wherein clientrequests of particular data packets to particular serving peers is madebased on a client analysis of availability vectors retrieved from eachof the serving peers, and wherein the availability vector received fromeach serving peer defines at least available data packets stored on eachcorresponding serving peer.
 8. The computer-readable medium of claim 1further comprising decoding and rendering the assembled multimedia datapackets on the client computer to provide to provide streaming mediaplayback on the client computer.
 9. The computer-readable medium ofclaim 1 wherein the data packets are embedded coded data packets. 10.The computer-readable medium of claim 9 further comprising maintainingclient control over a streaming bitrate of each embedded coded datapacket by automatically limiting the client requests of one or more datapackets to one or more of the serving peers as a function of: anaggregate serving bandwidth of the serving peers; a size of the commonstaging queue; a desired request fulfillment time; a length of receivedpackets in the common staging queue; a length of outstanding replies ineach request queue; and a base layer bitrate of the embedded coded datapackets.